Adapting physical activities and exercises based on physiological parameter analysis

ABSTRACT

Systems and methods for adapting physical activities and exercises based on analysis of physiological parameters are disclosed. A method includes: identifying, by a computer device, a user; receiving, by the computer device, data of the user while the user is engaged in exercise or physical activity; analyzing, by the computer device, the data to determine a detected state of the user; and providing, by the computer device, feedback to the user based on the analyzing the data.

BACKGROUND

The present invention generally relates to monitoring and analyzingphysical activities such as exercise and, more particularly, to a systemand method for adapting physical activities and exercises based onfacial analysis by image processing.

Exercise and physical activities have been demonstrated as being usefulfor a person to maintain healthy habits, decrease the risk of certaindiseases, avoid injuries, and help to feel better physically andmentally. However, exercising and performing physical activities withoutproper orientation can have the opposite effect, being occasionallyharmful to the person.

While a person is exercising, it is not trivial to assess whether theeffort level employed by the person is commensurate with what has beenprescribed by a professional (e.g., a medical professional or trainerwho prescribes certain exercise) and/or is within health limits of theperson. This is because the effort level employed by the person may bebased on a combination of plural variables that may be manifested indifferent physiological parameters. Thus, sometimes, not even the personwho is exercising is able of measuring their true effort level, and thismay result in an inefficient exercise or may even cause injuries.

SUMMARY

In a first aspect of the invention, there is a method that includes:identifying, by a computer device, a user; receiving, by the computerdevice, video data of a face region of the user while the user isengaged in exercise or physical activity; analyzing, by the computerdevice, the video data to determine a detected state of the user,wherein the analyzing the video data comprises performing facialanalysis using the video data; and providing, by the computer device,feedback to the user based on the analyzing the video data.

In another aspect of the invention, there is a system that includes: aCPU, a computer readable memory and a computer readable storage mediumassociated with a computing device; program instructions to receivevideo data of a face region of the user while the user is engaged inexercise or physical activity; program instructions to analyze the videodata using facial analysis to determine a detected state of the user;and program instructions to provide feedback to the user based oncomparing the detected state to a prescribed training rule. The programinstructions are stored on the computer readable storage medium forexecution by the CPU via the computer readable memory.

In another aspect of the invention, there is a computer program productthat includes a computer readable storage medium having programinstructions embodied therewith. The program instructions are executableby a computing device to cause the computing device to: receive videodata of a face region of the user while the user is engaged in exerciseor physical activity; analyze the video data using facial analysis todetermine a state of the user based on facial data of the user; andprovide feedback to the user based on the analyzing the video

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is described in the detailed description whichfollows, in reference to the noted plurality of drawings by way ofnon-limiting examples of exemplary embodiments of the present invention.

FIG. 1 depicts a computing infrastructure according to an embodiment ofthe present invention.

FIG. 2 shows a block diagram of an exemplary system in accordance withaspects of the invention.

FIG. 3 shows a flowchart of a method in accordance with aspects of theinvention.

DETAILED DESCRIPTION

The present invention generally relates to monitoring and analyzingphysical activities such as exercise and, more particularly, to a systemand method for adapting physical activities and exercises based onfacial analysis by image processing. According to aspects of theinvention, image processing is used to provide real-time feedback to auser when the user is performing physical activities such as exercising.In embodiments, video data of a face region of the user is obtainedwhile the user is exercising, and real time feedback is provided to theuser based on analyzing the video data. The analysis of the video datamay include determining a detected state of the user and comparing thedetected state to a user data. The detected state may include at leastone from the group consisting of: facial expression (e.g., tired, bored,calm, level of exertion, level of pain); perspiration detection;breathing interval estimation; and skin tone change. The feedback mayinclude at least one from the group consisting of: displaying anindication of the detected state; displaying an indication of the userdata; displaying a recommendation; and automatically adjusting anelement of the user's environment. In this manner, implementations ofthe invention provide a user with real time feedback about theirexercising so that the user may make an informed decision about how tocontinue or modify the exercising.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Referring now to FIG. 1, a schematic of an example of a computinginfrastructure is shown. Computing infrastructure 10 is only one exampleof a suitable computing infrastructure and is not intended to suggestany limitation as to the scope of use or functionality of embodiments ofthe invention described herein. Regardless, computing infrastructure 10is capable of being implemented and/or performing any of thefunctionality set forth hereinabove.

In computing infrastructure 10 there is a computer system (or server)12, which is operational with numerous other general purpose or specialpurpose computing system environments or configurations. Examples ofwell-known computing systems, environments, and/or configurations thatmay be suitable for use with computer system 12 include, but are notlimited to, personal computer systems, server computer systems, thinclients, thick clients, hand-held or laptop devices, multiprocessorsystems, microprocessor-based systems, set top boxes, programmableconsumer electronics, network PCs, minicomputer systems, mainframecomputer systems, and distributed cloud computing environments thatinclude any of the above systems or devices, and the like.

Computer system 12 may be described in the general context of computersystem executable instructions, such as program modules, being executedby a computer system. Generally, program modules may include routines,programs, objects, components, logic, data structures, and so on thatperform particular tasks or implement particular abstract data types.Computer system 12 may be practiced in distributed cloud computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed cloudcomputing environment, program modules may be located in both local andremote computer system storage media including memory storage devices.

As shown in FIG. 1, computer system 12 in computing infrastructure 10 isshown in the form of a general-purpose computing device. The componentsof computer system 12 may include, but are not limited to, one or moreprocessors or processing units (e.g., CPU) 16, a system memory 28, and abus 18 that couples various system components including system memory 28to processor 16.

Bus 18 represents one or more of any of several types of bus structures,including a memory bus or memory controller, a peripheral bus, anaccelerated graphics port, and a processor or local bus using any of avariety of bus architectures. By way of example, and not limitation,such architectures include Industry Standard Architecture (ISA) bus,Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, VideoElectronics Standards Association (VESA) local bus, and PeripheralComponent Interconnects (PCI) bus.

Computer system 12 typically includes a variety of computer systemreadable media. Such media may be any available media that is accessibleby computer system 12, and it includes both volatile and non-volatilemedia, removable and non-removable media.

System memory 28 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) 30 and/or cachememory 32. Computer system 12 may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 34 can be provided forreading from and writing to a nonremovable, non-volatile magnetic media(not shown and typically called a “hard drive”). Although not shown, amagnetic disk drive for reading from and writing to a removable,non-volatile magnetic disk (e.g., a “floppy disk”), and an optical diskdrive for reading from or writing to a removable, non-volatile opticaldisk such as a CD-ROM, DVD-ROM or other optical media can be provided.In such instances, each can be connected to bus 18 by one or more datamedia interfaces. As will be further depicted and described below,memory 28 may include at least one program product having a set (e.g.,at least one) of program modules that are configured to carry out thefunctions of embodiments of the invention.

Program/utility 40, having a set (at least one) of program modules 42,may be stored in memory 28 by way of example, and not limitation, aswell as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. Program modules 42 generally carry out the functions and/ormethodologies of embodiments of the invention as described herein.

Computer system 12 may also communicate with one or more externaldevices 14 such as a keyboard, a pointing device, a display 24, etc.;one or more devices that enable a user to interact with computer system12; and/or any devices (e.g., network card, modem, etc.) that enablecomputer system 12 to communicate with one or more other computingdevices. Such communication can occur via Input/Output (I/O) interfaces22. Still yet, computer system 12 can communicate with one or morenetworks such as a local area network (LAN), a general wide area network(WAN), and/or a public network (e.g., the Internet) via network adapter20. As depicted, network adapter 20 communicates with the othercomponents of computer system 12 via bus 18. It should be understoodthat although not shown, other hardware and/or software components couldbe used in conjunction with computer system 12. Examples, include, butare not limited to: microcode, device drivers, redundant processingunits, external disk drive arrays, RAID systems, tape drives, and dataarchival storage systems, etc.

FIG. 2 shows a block diagram of an exemplary system 100 in accordancewith aspects of the invention. In embodiments, the system 100 includes arecording device 105, an analysis server 110, a notification device 115,and an input device 120. The devices 105, 110, 115, 120 of the system100 may be operatively connected to a communication network 125 thatsupports computer-based communication between computer devices. Thenetwork 125 may be any desired network or combination of networks,including but not limited to: LAN, WAN, and the Internet. The devices ofthe system 100 may communicate with one another using, for example,application programming interface (API) calls or other suitable methodsof communication.

According to aspects of the invention, the recording device 105comprises a video camera that is configured to capture video data of aface region of a user that is exercising or performing other physicalactivity. The recording device 105 may be physically attached toequipment 128 (e.g., a treadmill, stationary bicycle, rowing machine,weight machine, etc.) on which the user is exercising, such that thefield of view of the recording device 105 includes a face region of theuser exercising on the equipment. Alternatively, the recording device105 may be attached to a ceiling, wall, floor or other structure thatprovides the recording device 105 a field of view that includes a faceregion of the user who is exercising. Alternatively, the recordingdevice 105 may be attached to moveable object, such as an unmannedaerial vehicle, that provides the recording device 105 a field of viewthat includes a face region of the user who is exercising. The recordingdevice 105 may comprise any suitable type of device that includes avideo camera, such as a stand-alone camera, a smartphone, or an unmannedaerial vehicle onboard camera. In embodiments, the recording device 105transmits the video data to the analysis server 110 via the network 125.

According to aspects of the invention, the analysis server 110 runs ananalysis tool 130. The analysis server 110 may be a computer system 12and the analysis tool 130 may be a program module 42 described withrespect to FIG. 1. In embodiments, the analysis server 110 running theanalysis tool 130 may perform one or more of the functions describedherein, including: receive video data from the recording device 105;analyze the video data to determine a detected state of the user;compare the detected state to user data; and provide feedback to theuser based on the comparison of the detected state to the user data.

As described herein, the analysis tool 130 is configured to analyze thevideo data (received from the recording device 105) to determine adetected state of the user that is currently exercising. The detectedstate may include at least one from the group consisting of: facialexpression (e.g., tired, bored, calm, level of exertion, level of pain);perspiration detection; breathing interval estimation; and skin tonechange.

The detected state may be facial expression, in which case the analysistool 130 may apply image processing techniques to the video data, inparticular facial recognition programming and/or facial analysisprogramming, to determine whether the user is tired, bored, calm,experiencing a certain level of exertion, or experiencing a certainlevel of pain while exercising. For example, the analysis tool 130 mayuse facial recognition/analysis techniques to compare the user's facialexpression (captured in the video data) to a database of categorizedfacial expressions to determine that the user's facial expression mostclosely matches a category of facial expression, e.g., tired, bored,calm, low exertion, medium exertion, high exertion, no pain, low pain,medium pain, high pain. Specifically, the analysis tool 130 may use deeplearning or action unit (AU) detection in conjunction with the databaseof categorized facial expressions to determine a category of the user'sfacial expression. Aspects of the invention are not limited to theseexemplary facial expressions, and other facial expressions may be usedin implementations. In this manner, the analysis tool 130 may beconfigured to determine, in real time, a category of facial expressionof the user while the user is exercising.

The detected state may be perspiration detection, in which case theanalysis tool 130 may apply image processing techniques to the videodata to determine whether the user is perspiring while exercising. Forexample, the analysis tool 130 may use image processing techniques tocompare an image of the skin texture of the user (captured in the videodata) to a database of categorized images of skin textures thatcorrespond to one of a perspiring state and a non-perspiring state. Inthis manner, the analysis tool 130 may be configured to determine, inreal time, whether or not the user is perspiring while exercising.

The detected state may be breathing interval estimation, in which casethe analysis tool 130 may apply image processing techniques to the videodata to determine a breathing interval of the user while exercising. Forexample, the analysis tool 130 may use image processing techniques totrack the changing shape of the user's mouth and nose to determine whenthe user is inhaling and exhaling, e.g., by comparing images of theuser's mouth and nose to a database of categorized images of mouth andnose that correspond to one of an inhaling state and an exhaling state.Additionally, by analyzing times associated with the video images of theuser's mouth and nose, the analysis tool 130 may determine an amount oftime that elapses between each determined inhale and exhale of the user.In this manner, the analysis tool 130 may be configured to determine, inreal time, a breathing interval (e.g., number of breaths per minute) ofthe user while exercising.

The detected state may be skin tone change, in which case the analysistool 130 may apply image processing techniques to the video data todetermine that the user's skin tone has changed from a first tone to asecond tone user while exercising. A change in skin tone may indicate,for example, that the user's skin is flush due to physical exertion. Forexample, the analysis tool 130 may use image processing techniques toobserve the user's initial skin tone at an early stage of the exercisingand detect if the user's skin tone changes from the initial skin tone toa different skin tone at a later stage of the exercising. In thismanner, the analysis tool 130 may be configured to determine, in realtime, a skin tone change of the user while exercising.

Still referring to FIG. 2, in embodiments, the analysis server 110 isconfigured to provide feedback to the user based on the detected stateof the user (as determined by the analysis tool 130). The feedback maybe based solely on the detected state of the user, or may be based oncomparing the detected state of the user to user data (e.g., dataassociated with the particular user). The feedback may include at leastone from the group consisting of: displaying an indication of thedetected state; displaying an indication of user data; displaying arecommendation; and automatically adjusting an element of the user'senvironment. The feedback, when comprising a display of information, maybe provided via the notification device 115, which may comprise a videodisplay device (e.g., an LCD screen of a television, computer,smartphone, etc.).

In an embodiment, the feedback may be based solely on the detected stateof the user, e.g., without comparing the detected state to the userdata. In this embodiment, the server 110 may transmit signals to thenotification device 115 to cause the notification device 115 to displayan indication 140 of the user's detected state (or plural detectedstates) in real time. In this manner, the user is provided withinformation that they may employ in deciding how to continue (or modify)their exercise.

For example, the server 110 may display an indication of the user'sdetected facial expression on the notification device 115, and changethe displayed indication in real time as changes in the user's facialexpression are detected by the server 110. As another example, theserver 110 may display an indication of the user's detected perspirationstate on the notification device 115, and change the displayedindication in real time as changes in the user's perspiration state aredetected by the server 110. As another example, the server 110 maydisplay an indication of the user's detected breathing interval (e.g.,breathing rate) on the notification device 115, and change the displayedindication in real time as changes in the user's breathing interval aredetected by the server 110. As another example, the server 110 maydisplay an indication of the user's detected skin tone on thenotification device 115, and change the displayed indication in realtime as changes in the user's skin tone are detected by the server 110.The server 110 may be configured to simultaneously display pluralindications of different detected states on the notification device 115,e.g., display detected facial expression, perspiration state, breathinginterval, and skin tone all at the same time on the notification device115.

In another embodiment, the feedback may be based on comparing thedetected state of the user to user data (e.g., data associated with theparticular user). The user data may comprise, for example, at least oneof: age, weight, height, gender, health conditions, and prescribedtraining. The user data may be defined by data that is input via theinput device 120, which may be a computer device that permits a user toenter data via any suitable method such as touch screen, keyboard,button(s), dial(s), and file transfer.

Prescribed training may be defined by the user, a medical professional(e.g., a doctor), or an exercise professional (e.g., trainer, coach,etc.). Prescribed training may define one or more thresholds for theuser in terms of possible detected states (e.g., tired, bored, calm, lowexertion, medium exertion, high exertion, no pain, low pain, mediumpain, high pain, perspiration, breathing interval, and skin tonechange). Prescribed training may be defined in terms of rules regardingone or more detected states that the user is recommended to achieveduring exercising. For example, an exemplary prescribed training rulemay define that a user is recommended to exhibit: medium or highexertion, perspiration, and a breathing interval of at least thirtybreaths per minute. Prescribed training may also be defined in terms ofrules regarding one or more detected states that the user is recommendedto avoid during exercising. For example, an exemplary prescribedtraining rule may define that a user is recommended to not exhibit: highpain or a breathing interval exceeding fifty breaths per minute. Theseexamples are not intended to be limiting, and any desired configurationof states may be defined in one or more prescribed training rules.Moreover, a single user may have plural different prescribed trainingrules that are simultaneously applied by the analysis tool 130.

In embodiments, user data that is entered via the input device 120 istransmitted to the server 110 and used by the analysis tool 130 onlyduring a single exercise session. In such an embodiment, the server 110does not store the user data for use in subsequent exercise sessions.

In other embodiments, user data that is input via the input device 120is transmitted to the server 110 and stored in association with anidentity of the user that provided the data. In this manner, the userdata may be entered one time and then retrieved from storage duringfuture exercise sessions by the same user. For example, a user may loginvia the input device 120, and the server 110 may retrieve the user databased on the user identity determined from the user login. In anotherembodiment, alternatively to a user logging in via the input device 120,the server 110 may automatically determine an identity of a user basedon image analysis (e.g., facial recognition) of a video of the face ofthe user obtained via the recording device 105. In this manner, theserver 110 may automatically determine the identity of the user andobtain the user data based on the determined user identity.

Still referring to the embodiment in which the feedback is based oncomparing the detected state to the user data, the feedback may includea visual indication 140 of the detected state and a visual indication145 of a portion of the user data, such as a prescribed training rule,without a recommendation (e.g., displaying the detected state and arelated prescribed training rule simultaneously on the notificationdevice 115). In this manner, the user may use the feedback to make theirown determination as how whether and how to adjust their level ofexertion (e.g., stay the same, slow down, or speed up).

The feedback may optionally include a recommendation to the user basedon the comparison of the detected state and the user data. For example,the server 110 may display a recommendation 150 to the user (e.g., “slowdown” or “change the exercise” or “increase recovery time”) when adetected state exceeds a recommended state as defined in a prescribedtraining rule (e.g., a detected level of exertion/pain/breathinginterval exceeds a maximum level of exertion/pain/breathing intervaldefined in a prescribed training rule). Similarly, the server 110 maydisplay a recommendation to the user (e.g., “speed up” or “increaseeffort” or “decrease recovery time”) when a detected state is lower thana recommended state as defined in a prescribed training rule (e.g., adetected level of exertion is less than a minimum level of exertiondefined in a prescribed training rule). In this embodiment, the server110 may simultaneously display the visual indication 140 of the detectedstate and the visual indication 145 of the prescribed training alongwith the recommendation 150.

In further embodiments, the server 110 may automatically control anelement of the user's environment based on the comparison of thedetected state and the prescribed training. The element may comprise theequipment 128 and/or a climate control system 155 in the room where theequipment 128 is located. For example, when a detected level of exertionis greater than a prescribed level of exertion (e.g., as defined in aprescribed training rule), the server 110 may automatically perform oneof: decrease the resistance of a weight machine; decrease the speedand/or incline of a treadmill; decrease the resistance of a stationarybicycle elliptical machine, or rowing machine; and decrease atemperature of the room in which the user is located. Conversely, when adetected level of exertion is less than a prescribed level of exertion(e.g., as defined in a prescribed training rule), the server 110 mayautomatically perform one of: increase the resistance of a weightmachine; increase the speed and/or incline of a treadmill; increase theresistance of a stationary bicycle, elliptical machine, or rowingmachine; and increase a temperature of the room in which the user islocated. For example, the server 110 may be operatively connected to acontroller and/or actuator on the equipment 128 to send control signalsto the equipment 128 to automatically adjust the resistance, speed,and/or incline of the exercise machine. Similarly, the server 110 may beoperatively connected to a thermostat of a climate control system 155 ofthe room in which the user is located to send control signals to thethermostat to automatically adjust the temperature of the room.

Aspects of the invention are not limited to the aforementioned examplesof feedback, and other actions may be automatically performed based onthe comparison of the detected state and the user data. For example, theserver 110 may provide a recommendation via the notification device 115recommending that the user ingest liquid or change the room temperaturebased on the server 110 detecting that the user's perspiration level istoo high, e.g., has detected a state of perspiration for a predefinedamount of time. As another example, the server 110 may provide arecommendation via the notification device 115 recommending that theuser adjust their breathing behavior based on the server 110 detectingthat the user's breathing intervals are too short or too long during anexercise. In yet another example, the server 110 may provide arecommendation via the notification device 115 based on comparing thedetected state of the user to health data of the user, e.g., bydetermining very low blood glucose levels based on detecting tremors,dizziness, pallor, excessive cold sweat, and weakness.

In embodiments, the recording device 105, the notification device 115,and the input device 120 may be combined into a single device, such as acomputer system 160 (e.g., similar to computer system 12 of FIG. 1) thatcomprises an integrated video camera for recording video images, adisplay screen for displaying visual output, and an input system forreceiving user input. In an exemplary implementation, the computersystem 160 is physically connected to the equipment 128 (e.g., atreadmill, stationary bicycle, rowing machine, weight machine, etc.). Inanother exemplary implementation, the computer system 160 is physicallyconnected to a wall, ceiling, or floor in the vicinity of the equipment128. Alternatively to being combined in a single system, each of therecording device 105, the notification device 115, and the input device120 may be separate devices. Each respective one of the devices may bephysically connected to the equipment 128 or physically connected to awall, ceiling, or floor in the vicinity of the equipment 128.

In embodiments, the server 110 is operatively connected to a sensor 165,which may comprise one or more of: a heart rate sensor; a skintemperature sensor; a sleep quality sensor; an oxygen saturation sensor;and a galvanic skin response sensor. In this manner, the server 110determines a detected state of the user based on a combination of thevideo data from the recording device 105 and the data from the sensor165.

FIG. 3 depicts a method of adapting physical activities and exercisesbased on facial analysis by image processing in accordance with aspectsof the invention. The steps of the method may be performed in the systemof FIG. 2 and are described with reference to the elements described inFIG. 2.

Referring to FIG. 3, at step 300 the system 100 receives an input ofuser data. Step 300 may be performed in the manner described withrespect to FIG. 2, e.g., by a user, a medical professional, or anexercise professional providing input to the input device 120. The userdata may include, for example, at least one of: age, weight, height,gender, health conditions, and prescribed training.

At step 305, the system 100 identifies the user. In embodiments, theuser provides login information via the input device 120 and the server110 determines the user identity based on the login information, e.g.,as described with respect to FIG. 2. In other embodiments, the server110 automatically determines the user identity based on image analysisof a video of the face of the user obtained via the recording device105, e.g., as described with respect to FIG. 2.

At step 310, the system 100 gathers video data of the user's face (e.g.,facial data) while the user is exercising. Step 310 may be performed inthe manner described with respect to FIG. 2, e.g., by the recordingdevice 105 obtaining video data of the user and transmitting the videodata to the server 110 via the network 125.

At step 315, the system 100 determines a detected state of the userbased on analyzing the video data (e.g., facial data) from step 310.Step 315 may include analyzing the video data using facial analysis todetermine a state of the user based on facial data of the user. Step 315may be performed in the manner described with respect to FIG. 2, e.g.,by the server 110 analyzing the video data to determine at least one of:tired, bored, calm, low exertion, medium exertion, high exertion, nopain, low pain, medium pain, high pain, perspiration, breathinginterval, and skin tone change. Other states may detected by the server110 using additional image processing rules programmed in the analysistool 130 of the server 110. In a preferred embodiment, the analyzingincludes facial analysis image processing to determine a facialexpression of the user, and may additional include image processing todetect at least one of perspiration, breathing interval, and skin tonechange.

At step 320, the system 100 provides feedback to the user based on theanalyzing the video data from step 315. Step 320 may be performed in themanner described with respect to FIG. 2, e.g., by the notificationdevice 115 displaying at least one of an indication of the detectedstate, user data, and a recommendation. In an embodiment, the feedbackis not based on a comparison of the detected state to the user data, andthe feedback in this embodiment is the notification device 115displaying an indication of the detected state in real time (based ondata received from the server 110). In another embodiment, the feedbackis based on a comparison of the detected state to the user data, and thefeedback in this embodiment is the notification device 115 displaying atleast one of an indication of the detected state in real time, userdata, and a recommendation. The user data that may be displayed at step320 may include, for example, a prescribed training rule that is relatedto the detected state that is currently being displayed. Therecommendation that may be displayed at step 320 may include, forexample, a recommendation to adjust the user's exercise based oncomparing the detected state to the prescribed training rule.

The feedback at step 320 may optionally include the system 100automatically adjusting an element of the user's environment. Forexample, as described with respect to FIG. 2, the server 110 may sendcontrol signals to at least one of the equipment 128 and a climatecontrol unit 155 to automatically adjust an element of the user'senvironment based on the analyzing the video data from step 315.

In embodiments, a service provider, such as a Solution Integrator, couldoffer to perform the processes described herein. In this case, theservice provider can create, maintain, deploy, support, etc., thecomputer infrastructure that performs the process steps of the inventionfor one or more customers. These customers may be, for example, anybusiness that uses technology. In return, the service provider canreceive payment from the customer(s) under a subscription and/or feeagreement and/or the service provider can receive payment from the saleof advertising content to one or more third parties.

In still additional embodiments, the invention provides acomputer-implemented method, via a network. In this case, a computerinfrastructure, such as computer system 12 (FIG. 1), can be provided andone or more systems for performing the processes of the invention can beobtained (e.g., created, purchased, used, modified, etc.) and deployedto the computer infrastructure. To this extent, the deployment of asystem can comprise one or more of: (1) installing program code on acomputing device, such as computer system 12 (as shown in FIG. 1), froma computer-readable medium; (2) adding one or more computing devices tothe computer infrastructure; and (3) incorporating and/or modifying oneor more existing systems of the computer infrastructure to enable thecomputer infrastructure to perform the processes of the invention.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A method, comprising: identifying, by a computerdevice, a user; receiving, by the computer device, video data of a faceregion of the user while the user is engaged in exercise or physicalactivity; analyzing, by the computer device, the video data to determinea detected state of the user, wherein the analyzing the video datacomprises performing facial analysis using the video data; andproviding, by the computer device, feedback to the user based on theanalyzing the video data, wherein the detected state comprises at leastone from the group consisting of: tired, bored, calm, level of exertion,and level of pain; wherein the detected state additionally comprises atleast one from the group consisting of: perspiration detection;breathing interval estimation; and skin tone change; and wherein theproviding the feedback comprises the computer device causing anotification device to concurrently display: an indication of thedetected state; an indication of a prescribed training rule included inuser data; and a recommendation that is based on a comparison of thedetected state to the user data, wherein the indication of the detectedstate includes simultaneously displaying on the notification devicedetected facial expression, perspiration state, breathing interval, andskin tone.
 2. The method of claim 1, wherein the identifying the usercomprises: the computer device receiving login data from the user via aninput device; and the computer device determining an identity of theuser based on the login data.
 3. The method of claim 1, wherein theidentifying the user comprises the computer device determining anidentity of the user by performing facial recognition using the videodata.
 4. The method of claim 1, wherein the video data is received bythe computer device via network communication from a recording devicerecording the video data.
 5. The method of claim 1, wherein theproviding the feedback comprises the computer device causing thenotification device to display the indication of the detected state inreal time while the user is engaged in the exercise or the physicalactivity.
 6. The method of claim 1, wherein the providing the feedbackcomprises the computer device controlling equipment on which the user isengaged in the exercise or the physical activity.
 7. The method of claim1, wherein the providing the feedback comprises the computer deviceadjusting a temperature of a room in which the user is engaged in theexercise or the physical activity.
 8. The method of claim 1, wherein:the computer device receives the video data from a video cameraconnected to exercise equipment on which the user is exercising; theuser data includes a prescribed training rule; the computer devicedetermines, based on comparing the detected state to the prescribedtraining rule, that a detected level of exertion is less than aprescribed level of exertion; and the providing the feedback comprisesthe computer device controlling the exercise equipment to increase aresistance, speed, or incline of the exercise equipment.
 9. The methodof claim 1, wherein: the computer device receives the video data from avideo camera connected to exercise equipment on which the user isexercising; the user data includes a prescribed training rule; thecomputer device determines, based on comparing the detected state to theprescribed training rule, that a detected level of exertion is more thana prescribed level of exertion; and the providing the feedback comprisesthe computer device controlling the exercise equipment to decrease aresistance, speed, or incline of the exercise equipment.
 10. The methodof claim 1, wherein the indication of the detected state includessimultaneously displaying on the notification device detected facialexpression, perspiration state, breathing interval, and skin tone.
 11. Asystem, comprising: a CPU, a computer readable memory and a computerreadable storage medium associated with a computing device; programinstructions to receive video data of a face region of the user whilethe user is engaged in exercise on exercise equipment comprising one ofa treadmill, stationary bicycle, rowing machine, and weight machine;program instructions to determine a detected state of the user, whereinthe determining the detected state includes: analyzing the video data todetermine a detected facial expression of the user; and analyzing thevideo data to determine one of perspiration detection of the user,breathing interval estimation of the user, and skin tone change of theuser; and program instructions to provide feedback to the user based oncomparing the detected state to a prescribed training rule, wherein thecomputing device comprises a server that receives the video data from arecording device via a network, wherein the providing the feedbackcomprises the server transmitting data to a notification device via thenetwork, wherein the providing the feedback further comprises the servercausing the notification device to concurrently display: an indicationof the detected state; an indication of the prescribed training rule;and a recommendation based on comparing the detected state to theprescribed training rule; and wherein the program instructions arestored on the computer readable storage medium for execution by the CPUvia the computer readable memory, wherein the indication of the detectedstate includes simultaneously displaying on the notification devicedetected facial expression, perspiration state, breathing interval, andskin tone.
 12. The system of claim 11, wherein the providing thefeedback comprises the server causing the notification device to displaythe indication of the detected state in real time while the user isengaged in the exercise on the exercise equipment.
 13. The system ofclaim 11, wherein the providing the feedback comprises the servercontrolling the exercise equipment to adjust a resistance, speed, orincline of the exercise equipment.
 14. The system of claim 11, whereinthe providing the feedback comprises the computer device adjusting atemperature of a room in which the user is engaged in the exercise onthe exercise equipment.
 15. The system of claim 11, wherein theindication of the detected state includes simultaneously displaying onthe notification device detected facial expression, perspiration state,breathing interval, and skin tone.
 16. A computer program productcomprising a computer readable storage medium having programinstructions embodied therewith, the program instructions executable bya computing device to cause the computing device to: receive video dataof a face region of the user while the user is engaged in exercise orphysical activity on exercise equipment comprising one of a treadmill,stationary bicycle, rowing machine, and weight machine; determine adetected state of the user, wherein the determining the detected stateincludes: analyzing the video data to determine a detected facialexpression of the user; and analyzing the video data to determine one ofperspiration detection of the user, breathing interval estimation of theuser, and skin tone change of the user; and provide feedback to the userbased on the analyzing the video, wherein the providing the feedbackcomprises the server causing the notification device to concurrentlydisplay: an indication of the detected state; an indication of aprescribed training rule; and a recommendation based on comparing thedetected state to the prescribed training rule, wherein the indicationof the detected state includes simultaneously displaying on thenotification device detected facial expression, perspiration state,breathing interval, and skin tone.
 17. The computer program product ofclaim 16, wherein the providing the feedback comprises the serversending a signal to control equipment on which the user is engaged inthe exercise or the physical activity.
 18. The computer program productof claim 16, wherein the providing the feedback comprises the serversending a signal to adjust a temperature of a room in which the user isengaged in the exercise or physical activity on the exercise equipment.19. The computer program product of claim 16, wherein the indication ofthe detected state includes simultaneously displaying on thenotification device detected facial expression, perspiration state,breathing interval, and skin tone.